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Accumulating  baseline  data  on  the  defense  acquisition  system  is  essential  to 
gauging  just  how  successful  reform  efforts  have  been  so  far.  This  article 
delineates  the  first  step  in  that  process. 


Baselining,  in  and  of  itself,  or  as  a 
step  in  continuous  process  improve¬ 
ment,  has  become  an  accepted  mod¬ 
ern  management  technique.  Baselining 
attempts  to  describe  and  capture  the  level 
of  success  of  an  existing  system.  Then  the 
proposed  system  changes  are  applied  to 
the  existing  system.  The  changed  system 
should  show  a  large  enough  increase  in 
success  over  the  existing  system  to 
warrant  the  cost  and  other  expenses  of 
implementation. 

The  Department  of  Defense  (DoD) 
Acquisition  Reform  (AR)  Program  is  a 
series  of  changes  being  incorporated  into 
the  DoD  acquisition  system.  This  article 
is  an  ex  post  facto  attempt  to  baseline  the 
DoD  acquisition  system  prior  to  the 
introduction  of  reforms.  In  order  to  do 
that,  we  have  to  determine  the  effective 
date  of  the  changes  and  the  level  of  suc¬ 
cess  of  the  then-existing  DoD  acquisition 


system.  Data  is  available  to  allow  us  to 
do  just  that. 

The  only  point  of  this  article  is  to 
identify  and  describe  the  first  step  in  a 
three-step  process  called  variously 
benchmarking,  baselining,  or  part  of 
continuous  process  improvement: 

•  Step  1  is  to  identify  a  process,  proce¬ 
dure,  or  product  into  which  a  series  of 
changes  or  improvements  are  to  be 
incorporated.  Describe  the  current  pro¬ 
cess,  procedure  or  product  as  carefully 
as  possible  as  regards  its  current 
effectiveness  and  efficiency.  Establish 
a  date  for  this  baseline  of  the  existing 
system. 

•  Step  2  is  to  introduce  the  changes  or 
improvements  into  the  process, 
procedure  or  product. 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 
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•  Step  3  is  to  measure  the  effectiveness 
and  efficiency  of  the  changed  system, 
at  some  future  date. 

Discussion 


First  we  shall  establish  the  effective 
date  of  acquisition  reform.  Table  1  chro¬ 
nologically  lists  most  of  the  AR  policy 
guidance  and  other  major  events  of  the 
program.  It  is  important  to  note  that  we 
are  seeking  the  first  date  that  AR  policies 
could  be  considered  effective  in  the  field; 
that  is,  the  approximate  date  that  AR 
initiatives  began  being  implemented  in  a 
significant  number  of  program  manage¬ 
ment  offices,  and  other  field  acquisition 
organizations.  In  the  January/February 
1997  Program  Manager  article,  Doreen 
Harwood  states  that  “A  gap  of  as  much  as 
six  months  can  occur  between  the  time  a 
statute  or  policy  change  is  issued  before 
it  is  received  in  the  field”  (Harwood,  1997, 
P-41). 

Colleen  Preston,  who  was  the  desig¬ 
nated  change  agent  to  direct  the  acquisi¬ 
tion  reform  program,  was  appointed 
Deputy  Under  Secretary  of  Defense  for 
Acquisition  Reform  (DUSD[AR])  in  June 
1993.  This  could  be  considered  the  start 
of  acquisition  reform,  but  certainly  not  the 
date  that  it  first  became  effective  in  the 
field.  Preston  outlined  the  initial  efforts 
of  acquisition  reform  four  years  later 
(1997,  pp.  25-26).  I  have  inserted  the 
dates  of  each  action  mentioned  (Table  1). 

We  started  off  initially  with  fol¬ 
lowing  up  on  the  Section  800 
Panel  recommendations  [January 
1993]...  So  we  made  that  the 
initial  thrust.  For  the  first  year  we 


practically  did  nothing  but  focus 
on  that  legislative  effort  day-to- 
day... Then  we  started  working  the 
Process  Action  Team  (PAT)... We 
started  with  electronic  commerce 
because  that  was  critical... 
Then... Specification  and  Stan¬ 
dards  issue  [June  1994]  we  took 
on  as  our  second  PAT... Then  we 
looked  at... contract  administra¬ 
tion,  the  procurement  process 
and.. .oversight  and  review  of  the 
systems  acquisition  process 
[December  1994].  That  particular 
PAT  process  was  very  difficult 
because  it  focused  on  the  relation¬ 
ship  between  OSD  and  the 
Services. 

Other  milestones  include  the  May  10, 
1995,  memorandum  of  William  Perry, 
then  Secretary  of  Defense,  implementing 
the  integrated  product  team  (IPT)  concept 
within  DoD;  the  initiation  of  cost  as  an 
independent  variable  (CAIV)  in  Decem¬ 
ber  1995;  and  the  release  of  the  new  5000 
Series  acquisition  documents  in  Decem¬ 
ber  of  1996.  Several  other  Milestone  dates 
can  be  extracted  from  the  chronology,  but 
these  show  the  time  required  for  approach¬ 
ing  effective  AR  implementation  in  the 
field.  An  alphabetical  list  of  AR  initiatives 
is  in  Figure  1 . 

Throughout  the  AR  implementation 
period  there  was  recognition  that  for  this 
“cultural  change”  to  be  effective  there  had 
to  be  visible  and  continuous  support  from 
the  top  and  available  tools  to  understand 
what  was  desired.  Again,  the  chronology 
shows  us  the  many  initiatives  taken  along 
these  lines,  such  as:  Paul  Kaminski 
hosting  a  one-day  DoD  offsite  (“Institu¬ 
tionalizing  IPTs”)  on  July  20,  1995; 
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Table  1.  Chronology  of  Acquisition  Reform  Events 


Date 

Event 

February  1991 

DoDD  5000.1  DoDI  5000.2  changed  and  reissued  and  5000. 2M 
promulgated. 

January  1993 

The  Acquisition  Law  Advisory  Panel  (Section  800  Panel)  findings 
reported  to  Congress.® 

June  1993 

Colleen  Preston  assumes  the  position  as  Deputy  Under  Secretary 
of  Defense  for  Acquisition  Reform.® 

Octeber  1993 

Federal  Acquisition  Streamlining  Act  (FASA)  of  1994  enacted.® 

First  quarter  1994 

The  Advanced  Concept  Technology  Demonstration  program 
initiated.*’ 

February  1994 

William  J.  Perry  replaces  Les  Aspin  as  Secretary  of  Defense.® 

February  1994 

Secretary  Perry  Issues  “Acquisition  Reform,  A  Mandate  for 

Change.”® 

March  1994 

Secretary  Perry  attaches  “Mandate  for  Change”  to  a  letter  to  the 
leadership  of  the  Department  of  Defense.® 

June  1994 

Preston  authors  an  article,  “Acquisition  Reform — Making  it  a 

Reality,”  in  Phalanx:  the  Bulletin  of  Military  Operations  Research 
(June  1994,  27[2]).The  article  concludes  with  a  section  titled,  “Flow 
Can  You  Participate?”® 

June  1994 

Secretary  Perry  Issues  memo:  “Specifications  and  Standards — A 

New  Way  of  Doing  Business.”® 

Octeber  1994 

Paul  Kaminski  sworn  In  as  Under  Secretary  of  Defense  for  Acquisi¬ 
tion  and  Technology  (USD[A&T]).® 

c.  1994 

DUSD(AR)  position  to  report  to  USD(A&T).® 

December  1994 

The  Oversight  and  Review  of  the  Systems  Acquisition  Process  PAT 
report  published.*’ 

December  1994 

The  Defense  Acquisition  Pilot  Program  launched  as  allowed  by 

FASA. 

March  1995 

USD(A&T)  establishes  an  IPT  for  the  purpose  of  rewriting  the 
February  23,  1991,  5000  Series  documents.*’ 

April  1995 

Kaminski  Issues  a  memerandum,  “Reengineering  the  Acquisition 
Oversight  and  Review  Process.”  First  recommendatiens  of  the  PAT 
team  approved.*’ 

(continued) 
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Date 

Event 

May  1 995 

Secretary  Perry  implements  the  IPT  concept  for  DoD  via  a 
memorandum.® 

July  1995 

Kaminski  holds  a  DoD  offsite  entitled  “Institutienalizing  IPTs — 

DoD’s  Commitment  to  Change.”*’ 

November  1995 

Rules  ef  the  Road:  A  Guide  for  Leading  Successful  Integrated 

Product  Teams  is  published.*’ 

December  1995 

CAIV  was  initiated.® 

December  1995 

USD(A&T)  issues  guidance  for  making  “class  action”  contract 
changes  to  existing  contracts  on  a  facility-wide  basis.  AKA  Single 
Process  Initiative  (SPI).*’ 

February  1996 

DoD  Guide  to  Integrated  Product  and  Process  Development, 

(Version  1 .0)  issued  by  the  OUSD(A&T).® 

February  1996 

Director,  Test,  Systems  Engineering,  and  Evaluation,  publishes 

DoD  Guide  to  IPPD,  Version  1.0.*’ 

March  1996 

Update  of  the  DoD  5000  Documents  appreved  by  the  USD(A&T), 
DOT&E,  and  ASD  (C3I).® 

March  1996 

The  ODUSD(AR)  produces  the  video  The  Overarching  and 

Working  Level  Integrated  Product  Teams,  and  the  OIPT-WIPT 
Informatien  Guide.*’ 

April  1996 

DoD  and  Texas  Instruments  sign  first  SPI  agreement  for  manufac¬ 
turing  standards  for  all  its  products.*’ 

May  1996 

DoD  Acquisition  Reform  Day  is  held.*’ 

July  1996 

The  Defense  Acquisition  Deskbook,  first  piece,  released.*’ 

September  1996 

Kaminski’s  memerandum  provides  guidance  for  dealing  with  specifi- 
catien  or  process  changes  en  subcontracts  (SPI).*’ 

December  1996 

The  publishing  of  DoD  5000. 2R,  Mandatory  Procedures  for  Major 
Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated  Infor¬ 
mation  Systems  (MAIS)  Acquisition  Programs.  (Includes  change  1).® 

“  All  data  and  information  obtained  from  Defense  Systems  Management  College.  (1997,  December ).  A  Model  for 
Leading  Change:  Making  Acquisition  Reform  Work  (report  of  the  1996-1997  DSMC  Military  Research  Fellows). 
Fort  Belvoir,  VA:  Author. 

*’  All  data  and  information  obtained  from  Defense  Systems  Management  College.  (1 997,  January-February ).  Acqui¬ 
sition  reform  — the  end  of  the  beginning.  Program  Manager  (special  issue),  26(1). 
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Acquisition  Law  Advisory  Panel  (Section  800  Panel) 

Audit/Inspection  Reform 

Advanced  Concept  Technology  Demonstration  (ACTD) 

Buying  Commercial 

Defense  Acquisition  Deskbook 
Defense  Acquisition  Pilot  Programs  (DAPP) 

Direct  Vendor  Delivery 
Dual-Use  Technology 

Eleotronic  Commerce 
Empowerment 

Federal  Acquisition  Streamlining  Act  of  1994  (FASA) 

Integrated  Product  and  Process  Teams 

Lean  Logistics 

Modeling  and  Simulation 
Multiyear  Contracts 

Outsourcing 

Partnerships  with  Industry 
Performance-Based  Contracting 
Privatization 
Program  Stability 

Reduced  Government  Oversight 
Reengineering  the  Acquisition  Oversight  and  Review  Process 

Single  Process  Initiative  (SPI) 

Specifications  and  Standards  Policy 
Streamlined  Solicitation  Packages 

Update  and  Reissue  of  the  DoD  5000  Series  Documents 

Workforce  Education 

Figure  1.  Alphabetical  List  of  Acquisition  Reform  Initiatives 

publishing  Rules  of  the  Road:  A  Guide  From  ail  of  this  it  is  possible  to  esti- 
for  Leading  Successful  Integrated  Prod-  mate  when  AR  could  first  have  had  a 
uct  Teams  in  November  1995;  the  DoD  practical  effect  in  the  field.  For  me,  this 
AR  Day  of  May  31,  1996;  and  the  release  date  is  on  or  about  January  1996,  although 
of  the  Defense  Acquisition  Deskbook  in  efforts  continue  beyond  this  date,  and  will 
July  1996.  into  the  future.  January  1996  is  the 


37 


Acquisition  Review  Quarteriy—  Winter  2000 


estimated  date  that  a  sizable  number  of 
the  acquisition  workforce  in  the  field 
could  first  be  expected  to  put  acquisi¬ 
tion  reform  into  practice.  The  date  we 
are  more  interested  in,  however,  is  the  first 
probable  date  that  AR  could  affect  a  major 
defense  acquisition  program. 

Prior  research  has  shown  that  the  aver¬ 
age  length  of  an  Acquisition  Category  I 
(ACAT  I)  program  in  the  engineering  and 
manufacturing  development  (EMD)  phase 
is  7.4  years  (Gailey,  Reig,  and  Weber, 
1995).  The  Milestone  III  (MS  III)  deci¬ 
sion  generally  concludes  the  EMD  phase, 

where  design 
and  cost-im¬ 
pact  decisions 
determine 
more  than  80 
percent  of  the 
total  life-cycle 
system  costs. 
If  a  program’s 
MS  III  is  6 
months  away,  there  is  little  chance  AR 
changes  will  affect  that  program  in  EMD. 
Therefore,  if  acquisition  reform  first  be¬ 
came  effective  within  the  field  acquisition 
workforce  on  or  about  January  1996,  it 
could  only  have  an  EMD  effect  on  pro¬ 
grams  whose  MS  III  Defense  Acquisition 
Board  (DAB)  meeting  was  after  July 
1996.  The  MS  III  program  date  is  impor¬ 
tant  because  that  is  the  point  at  which  we 
will  measure  the  success  of  the  then- 
existing  DoD  acquisition  system. 

The  influence  of  two  new  initiatives  on 
the  effective  date  of  AR  for  any  particular 
program  has  not  been  discussed.  These 
initiatives  are  the  Advanced  Concept 
Technology  Demonstration  Program 
(ACTD),  and  the  Defense  Acquisition 
Pilot  Programs  (DAPPs),  both  initiated  in 


1994.  Ten  ACTDs  were  initiated  in  fiscal 
year  1995,  and  12  in  fiscal  year  1996.  This 
is  a  very  small  number  of  programs  when 
compared  to  the  approximately  200  pro¬ 
grams  listed  at  any  one  time  on  the  Office 
of  the  Secretary  of  Defense  Test  and 
Evaluation  (OSD  T&E)  program  over¬ 
sight  list.  Extracts  of  a  report  on  the 
lessons  learned  and  recommendations  on 
how  to  proceed  with  DoD -wide  imple¬ 
mentation  of  the  DAPP  initiative  are 
shown  in  Appendix  A.  An  exhaustive 
review  of  the  effects  of  DAPP  programs 
on  acquisition  efficiency  is  beyond  the 
scope  of  this  research,  however.  Other 
ongoing  Office  of  the  Secretary  of 
Defense  (OSD)  review  and  analysis  efforts 
explore  this  area  in  much  greater  depth. 

Although  not  considered  part  of  the 
acquisition  reform  program,  the  first  event 
listed  in  Table  1  is  worthy  of  discussion. 
In  1991,  two  significant  events  occurred 
that,  in  retrospect,  embodied  the  “cultural 
change”  so  central  to  acquisition  reform. 
The  first  of  these,  the  Eebruary  23,  1991, 
revision  to  the  5000  Series  documenta¬ 
tion,  modified  an  earlier  version  of  the 
5000  Series  that  required  important  user 
requirements  like  the  operational  require¬ 
ments  document  (ORD)  and  initial  opera¬ 
tional  capability  (IOC),  to  be  stated  at  MS 
I.  Prior  to  Eebruary  1991,  it  was  gener¬ 
ally  understood  that  these  requirements 
were  firm  and  not  subject  to  change.  In 
the  revision,  the  new  5000  Series  stated 
these  and  other  requirements  were  sub¬ 
ject  to  review  and  change  if  necessary  at 
each  Milestone.  This  allowed  for  a  more 
reasoned  approach  to  changing  require¬ 
ments  as  more  data  accumulated,  and 
allowed  the  program  manager  to  suggest 
changes  in  a  more  receptive  environment. 
Changing  user  requirements  if  necessary 


"In  1991,  two 
significant  events 
occurred  that,  in 
retrospect,  embod¬ 
ied  the  "cultural 
change"  so  central 
to  acquisition 
reform." 
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at  stages  in  the  development  of  a  system 
became  similar  to  the  latter-day  CAIV 
approach,  albeit  for  a  less  focused  reason. 

Also  in  1991,  the  commanders  of  the 
Service  operational  test  activities  (OTAs) 
realized  that  they  no  longer  could  operate 
in  the  mode  of  being  the  independent 
director  of  the  “final  exam,”  the  initial 
operational  test  and  evaluation  and  the 
operational  evaluation  (lOT&E/OPEVAL) 
just  prior  to  MS  III.  Rather,  they  initiated 
an  earlier  consultative  role  with  the 
developing  activity  and,  by  means  of  early 
operational  assessments,  worked  with  the 
program  managers  to  clarify  what  would 
be  expected  at  the  lOT&E/OPEVAE.  This 
change  in  modus  operandi  occurred  before 
the  AR  initiative  of  IPTs,  but  clearly 
achieves  some  of  the  same  desirable 
objectives. 

Because  of  these  two  activities,  the 
introduction  of  a  “forerunner  CAIV”  and 
a  “forerunner  IPT,”  the  year  1991  could 
be  dubbed  the  year  of  the  unheralded  AR. 
Because  this  was  a  revision  to  the  estab¬ 
lished  5000  Series  system,  the  lead  time 
to  be  effective  in  the  field  might  be  half 
that  required  for  the  later,  entirely  new, 
more  expansive  AR  changes. 

With  the  above  rationale,  the  first  pro¬ 
grams  that  would  be  beneficially  affected 
by  AR  would  be  those  whose  MS  III  DAB 
occurred  in  July  1996  or  afterward.  There¬ 
fore,  the  baseline  of  interest  would  be  the 
success  level  of  programs  whose  MS  III 
DAB  was  prior  to  July  1996.  This  was 
the  success  level  of  the  DoD  acquisition 
system  prior  to  AR  changes.  To  measure 
program  success,  we  will  use  the  standard 
parameters  of  cost,  schedule,  and  perfor¬ 
mance  of  a  program  during  EMD.  Both 
cost  and  schedule  successes  are  obtained 
by  a  review  of  a  program’s  selected 


acquisition  reports  (SARs)  during  EMD, 
and  performance  success  will  be  based  on 
a  review  of  the  program’s  lOT&E/ 
OPEVAE  test  reports  and  the  associated 
beyond  low-rate  initial  production 
(BERIP)  evaluation  issued  by  the  Direc¬ 
tor,  Operational  Test  and  Evaluation 
(DOT&E). 

Eortunately,  these  data  have  been  ob¬ 
tained  for  a  different  purpose  over  the  past 
four  years.  A  technical  report  has  been 
published  de¬ 
tailing  the  con¬ 
cept  and  ratio¬ 
nale  behind  the 
data  obtained  to 
date  (Gailey, 

Reig,  and  We¬ 
ber,  1995,  pp. 

3. 2-3. 3).  Eor 
each  of  the  pa¬ 
rameters — cost, 
schedule,  and 
performance — 
we  have  assigned  a  success  rating  of  from 
5  (very  good)  to  1  (poor).  The  cost  and 
schedule  data  from  the  Blue  Books  (no 
longer  maintained)  or  the  SARs  are 
objective,  using  the  DoD  standard  decre¬ 
ments  of  15  percent  in  cost  and  6  months 
in  schedule.  Performance  success  ratings 
were  subjectively  assigned  using  descrip¬ 
tive  criteria  that  delineated  between  the 
five  possible  ratings.  Operational  test 
reports  addressed  operational  effective¬ 
ness  and  operational  suitability.  Eor  our 
research  purposes,  we  have  assigned  a 
third  rating,  overall  operational  success, 
which  results  in  three  performance  suc¬ 
cess  rating  from  the  Service  OTA  and 
three  from  the  DOT&E  on  the  OSD  staff. 
In  practice,  the  Service  OTA  conducts  the 
operational  test,  but  by  law,  the  DOT&E 


"To  measure 
program  success, 
we  will  use  the 
standard  parameters 
of  cost,  schedule, 
and  performance  of 
a  program  during 
engineering  and 
manufacturi  ng 
development." 
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must  provide  an  independent  evaluation 
of  the  test  adequacy  and  the  operational 
effectiveness  and  suitability  of  the  system 
under  test. 

For  other  purposes,  we  have  gathered 
EMD  cost,  schedule,  and  performance 
data  on  programs  during  the  years  1980 
to  1996.  We  are  in  the  process  of  obtain¬ 
ing  the  same  data  for  programs  as  they 
complete  their  EMD  development  phase 
from  1996  on.  The  resultant  success 
ratings  are  shown  in  Table  2,  and  the 
percentage  equivalents  of  the  success 
ratings  are  shown  in  Table  3.  We  have 
listed  two  periods  from  which  to  choose 
the  performance  success  level  of  programs 
in  EMD.  The  first  line  in  Table  2  has  the 
success  ratings  for  all  programs  in  the  data 
base  from  1980  to  July  1996.  The  second 
line  shows  the  results  for  the  4  years  prior 
to  the  date  AR  could  have  influenced 
systems  in  EMD.  The  data  base  allows 
other,  different  periods  to  be  used. 


Conclusions 


Here  we  have  looked  only  at  the  cost, 
schedule,  and  performance  success  ratings 
of  programs  whose  EMD  phase  ended 
prior  to  July  1996.  After  this  date,  AR 
efforts  began  to  have  an  effect  on  major 
defense  acquisition  programs.  This  estab¬ 
lished  the  performance  level  of  the  DoD 
(pre-acquisition  reform)  acquisition  sys¬ 
tem.  The  improvements  in  the  acquisition 
system  due  to  ARs  can  be  measured  in  a 
few  years  using  a  similar  methodology.  It 
is  probably  too  early  to  attempt  to  measure 
the  improvements  in  cost,  schedule,  and 
performance  success  of  the  weapon 
systems  being  procured  by  the  DoD 
acquisition  system.  But  if  the  research 
database  established  at  Defense  Systems 
Management  College  continues  to  grow, 
in  the  future  such  data  will  be  at  hand. 

We  have  not  attempted  to  analyze  any 
of  the  many  other  parameters  of  AR,  such 
as  value  added  due  to  a  better  educated 
workforce,  etc.  Eigure  1  lists  all  or  almost 
all  of  the  initiatives  considered  to  be  part 


Table  2. 

Average  Baseline  Success  Ratings  -  Acquisition  Reform  Baseline 


M/S  III 

lOT&E/OPEVAL  Resuits 

DOT&E  BLRIP  Evaluation 

Time  Period 
and  Number 
of  Programs 

Cost 

Success 

Cost 

Percent 

Overrun 

Scheduie 

Success 

Scheduie 

Percent 

Overrun 

Effectiveness  Sultabiiity  Overail 

Effectiveness  Suitability  Overall 

1980-Jul  96 
(n=42) 

3.6 

31% 

2.6 

64% 

4.0  3.7  4.0 

3.8  3.6  3.8 

Jul  92-Jul  96 

3.5 

46% 

2.1 

81% 

4.3  3.9  4.2 

3.7  3.7  3.8 

Post  Acquisition  Reform  Success 

Aug  96-ON 

• 

• 

• 

• 

. 

. 

*  =  To  be  determined 
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Table  3. 

DSMC  EMD  Research  -  Success  Ratings  and  Percent  Equivalents 

Success  Rating 

Percent  (%) 

5  (very  good)  -  1  (poor) 

Equivalents 

5.0 

100 

4.8 

96 

4.6 

92 

4.4 

88 

4.2 

84 

4.0 

80 

3.8 

76 

3.6 

72 

3.4 

68 

3.2 

64 

3.0 

60 

2.8 

56 

2.6 

52 

2.4 

48 

2.2 

44 

2.0 

40 

1.8 

36 

1.6 

32 

1.4 

28 

1.2 

24 

1.0 

20 

of  the  AR  program  and  may  be  of  some 
interest  in  itself.  There  are  other  OSD 
analyses  in  existence  that  are  much 
broader  in  scope  and  level  of  effort; 
these  give  the  reader  a  broader  look  at  AR 
improvements. 

However,  using  the  results  achieved  by 
the  pre-AR  DoD  acquisition  system  in  the 
4  years  before  AR  began,  and  converting 
the  success  ratings  in  Table  2  to  percent 


equivalents  (Table  3),  schedule  success — 
approximately  42  percent  successful  in  the 
old  DoD  acquisition  system — could  cer¬ 
tainly  be  improved  by  AR  efforts.  Cost 
success  at  70  percent  should  also  be  ame¬ 
nable  to  improvement.  Future  improve¬ 
ment  in  performance  success,  where  the 
overall  success  level  was  76-84  percent, 
may  be  more  challenging. 
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Appendix  A 


Review  of  Defense  Acquisition 
Pilot  Programs 


The  Defense  Acquisition  Pilot  Pro¬ 
grams  (DAPP)  was  initiated  as  a  result  of 
the  Federal  Acquisition  Streamlining  Act 
(FAS A)  of  1994,  which  authorized  the 
DoD  to  experiment  with  new  ways  of 
doing  business  to  achieve  the  ohjective  of 
buying  systems  better,  faster,  and  cheaper. 
Seven  DAPPs  are  reported  on  and  data 
are  being  collected  on  what  aspects  of  the 
program  are  successful,  and  by  how  much 
(metrics).  Six  of  the  seven  programs  are: 
Commercial  Derivative  Engine,  Defense 
Personnel  Support  Center,  Hercules  (C- 
130J),  Joint  Primary  Aircraft  Training 
System,  Fire  Support  Combined  Arms 
Tactical  Trainer,  and  Joint  Direct  Attack 
Munitions.  This  exhibit  contains  my  im¬ 
pressions  of  data  whose  source  is  the  1996 
Compendium  of  Pilot  Program  Reports 
issued  by  the  Pilot  Program  Consulting 
Group. 

The  Commercial  Derivative  Engine 

(CDE)  is  produced  by  Pratt  and  Whitney 
and  used  on  the  C-17  aircraft.  In  1980, 
McDonnell  Douglas  Aircraft  Company 
selected  the  engine  to  power  the  C-17, 
and  thus  the  program  was  well  under  way 
when  it  was  nominated  in  July  1993  to  be 
a  pilot  program  to  “demonstrate  the 
advantages  of  using  derivatives  of 
commercial  engines  to  satisfy  military 
requirements.”  This  program  may  best 
exemplify  the  “buy  commercial”  initiative 
of  acquisition  reform. 

The  Defense  Personnel  Support 
Center  (DPSC)  began  reengineering  its 
business  practices  in  1989,  and  the 


designation  as  a  pilot  program  facilitated 
further  expansion  of  these  initiatives. 

Continued  participation  as  a  regu¬ 
latory  DAPP  allows  evaluation  of 
statutory  relief  provided  commer¬ 
cial  item  acquisition  by  EASA.... 

The  1995  report  highlighted  the 
fact  that  regulatory  relief  by  it¬ 
self  was  insufficient  to  make  a 
dramatic  impact....  It  is  the  acqui¬ 
sition  reform  environment  itself 
that  has  contributed  the  most  to 
the  continued  success... of  these 
initiatives. 

This  pilot  program  may  exemplify  the 
difficult-to-measure  improvements  result¬ 
ing  from  a  DoD  acquisition  “cultural 
change.” 

The  Hercules  (C-130J)  was  desig¬ 
nated  a  DAPP  program  in  September  1995 
and  “serves  as  the  first  major  procurement 
that  can  draw  upon  the  new  commercial 
practices  implemented  by  EASA.”  This 
ACAT  II  program  has  developed  a  com¬ 
prehensive  set  of  specific  and  bridge 
metrics  for  overall  DAPP  program  guid¬ 
ance.  These  metrics  primarily  address 
business  practice  issues,  but  contain  two 
performance  metrics.  The  business  prac¬ 
tice  metrics  appears  to  have  been  achieved 
very  nicely,  but  the  two  performance 
objectives  are  to  be  determined. 

The  Joint  Primary  Aircraft  Training 
System  (JPATS)  is  an  ACAT  IC  program 
whose  9-year  engineering  management 
and  development  phase  contract  was 
signed  in  Eebruary  1996.  “JPATS  specific 
metrics  were  developed... to  reflect  the 
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unique  commercial  aspects  of  the 
program.. .and  include: 

•  regulatory  and  statutory  relief, 

•  program  costs, 

•  RFP  preparation  and  content, 

•  funding  stability, 

•  ground-based  training  system, 

•  earned  value  reporting  system, 

•  program  office  staffing, 

•  contractor  team  composition,  and 

•  contract  administrative  services.” 

Again,  these  metrics  primarily  address 
business  aspects.  The  two  performance 
bridge  metrics  used  are  Anthropological 
Accommodation,  and  Birdstrike  Capabil¬ 
ity  @  270KTAS. 

The  Fire  Support  Combined  Arms 
Tactical  Trainer  (FSCATT)  RFP  was 
designed  to  incorporate  DAPP  acquisition 
reform  initiatives....  The  contract,  fixed- 
price,  7 -year  period  of  performance  was 
awarded  in  June  1995.  “As  a  DAPP,  [the 
program]  is  intended  to  demonstrate  that 
the  concepts  of  dual-use  technology  could 
be  applied  to  a  defense  program  in  addi¬ 
tion  to  demonstrating  the  capability  of 
integrating  commercial  and  nondevelop- 
mental  item  components  into  a  complete 
system.”  This  is  an  ACAT  III  program. 
The  program  metrics  consist  entirely  of 
business  practice  parameters,  with  no  per¬ 
formance  metrics,  except  for  the  number 
of  work  hours  for  quality  assurance  and 


test  and  evaluation.  Business  metric 
results  to  date  are  good,  with  cost  sav¬ 
ings  estimated  at  13.5  percent  and  a 
planned  schedule  reduction  of  33  percent. 

The  Joint  Direct  Attack  Munitions 
(JDAM)  may  be  the  most  discussed  and 
closely  followed  DAPP  program  to  date. 

The  nature  of  JDAM  suggests  that 
the  major  savings. ..would  accrue 
in  production  rather  than  devel¬ 
opment....  The  [program]  is  in¬ 
tended  to  demonstrate  the  prime 
contractor  and  key  subcontractors 
are  able  to  develop  the  JDAM 
using  practices,  ...from  their  com¬ 
mercial  sector  business  base.... 
Finally,  the  program  is  expected 
to  meet  the  planned  development 
schedule  without  the  process 
delays  that  have  been  incurred  on 
other  major  defense  programs. 

JDAM  is  an  ACAT  I  program  whose 
Phase  I  EMD  effort  began  in  April  1994. 
Phase  II  EMD  started  in  October  1995 
with  the  down- select  to  one  contractor. 
The  Phase  II  (EMD)  contract  is  CPEE 
with  a  period  of  performance  of  approxi¬ 
mately  3  1/2  years.  “Thus,  the  cultural 
change  provided  by  pilot  programs 
designation  had  the  greatest  impact  on 
JDAM  and  resulted  in  the  implementa¬ 
tion  of  [several  new]  business  practices.” 
The  cost  metric  includes  total  program 
costs,  but  Table  1.3.2  shows  Phase  II  EMD 
(RDT&E)  cost  avoidance’s  to  total 
$49. 8M,  including  $7.3M  attributable  to 
reduced  wind  tunnel  tests,  $30M  of 
reduced  open  air  test  A/C  and  test  units, 
and  $12M  due  to  schedule  acceleration. 

In  discussing  program  operational  per¬ 
formance  and  cost,  the  report  states 
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“...large  reductions  in  unit  price  were 
possible  by  trading  other  ‘performance’ 
requirements.”  Two  meaningful  perfor¬ 
mance  metrics — end  game  accuracy  and 
reliability — are  being  tracked.  The  JDAM 
program  1996  status  report  concludes: 


The  most  dramatic  results  of  the 
JDAM  acquisition  reform  efforts 
are  the  reported  reduction  in  to¬ 
tal  program  costs  of  $2.96  billion 
Then  Year  (TY)  over  a  10-year 
production  cycle,  a  39  percent 
reduction  in  contract  administra¬ 
tion  hours  to  date,  and  a  35  per¬ 
cent  reduction  in  development 
time.  Clearly  JDAM  results  to 
date  demonstrate  the  applicabil¬ 
ity  of  commercial  practices  and 
other  innovative  management 
practices  to  major  defense  acqui¬ 
sition  programs  and  the  efficiency 
gains  that  can  be  achieved.  As 
reported  by  the  program  office, 
acquisition  reform  combined  with 
common-sense  management  is 


enabling  JDAM  to  realize  sub¬ 
stantial  in-house  efficiency  gains, 
reduce  contract  costs,  and 
improve  cycle  times. 

Conclusions 

Have  these  Defense  Acquisition  Pilot 
Programs  had  an  effect  on  the  selection 
of  the  cost,  schedule,  and  performance 
baseline  of  the  pre-AR  DoD  acquisition 
system?  I  believe  not,  since  two  of  the 
DAPPs  pre-date  acquisition  reform  con¬ 
siderably,  three  are  not  AC  AT  I  programs, 
one  contains  possibly  minor  performance 
metrics,  and  three  contain  no  performance 
metrics  at  all.  By  contrast,  the  entire  thrust 
of  this  article  is  the  combined  measure¬ 
ment  of  cost  and  schedule  data  (from 
SARs),  and  performance  data  (from 
operational  testing  reports)  of  ACAT  I 
programs  in  EMD  prior  to  the  effective 
start  date  of  acquisition  reform.  A  more 
reasonable  comparison  would  be  DAPP 
program  performance  as  measured  by 
other  “standard”  programs  being  developed 
in  the  current  acquisition  reform  era. 
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